Analytic Information Theory and the Redundancy Rate Problem
نویسندگان
چکیده
One of the most basic problems of information theory 1] is that of source coding. A source is by deenition a mechanism that produces messages over a nite alphabet A, a message of length n being conventionally denoted by x n 1 = (x 1 ; : : : ; x n). A code C is a translation mechanism (an injective function, an algorithm) that, for each n, takes as input a message from A n and transforms it into a binary sequence. Such a translation is thus a xed-length to variable-length encoding. Messages have some structure. For the English language source, the sequence \Rzqxwa gkvzzxq wzd aaaaaaa rxbleurp" is much less likely than the sequence \It rained yesterday over England." Indeed, some letters are more frequent than others, certain letter combinations are impossible, etc. It is then customary to try and capture the principal features of the source by some probability distribution of sorts over A n. The main models considered in the talk are the following. M1. A memoryless model considers letters as independent identically distributed random variables, with letter i 2 A having probability p i. (This is sometimes called the Bernoulli model.) M2. A Markov model assumes an underlying nite set of states with transition probability p i;j between states i and j and a mapping from states to letters. As discovered by Shannon around 1949, information is measured by entropy. The entropy of a probability distribution P = fp s g s2S over any nite set S is deened as H(P) := ? X s2S p s lg p s ; where lg x = log 2 x. (Roughly, the deenition extends the fact that an element in a set of cardi-nality m needs to be encoded by about lg m bits in order to be distinguished from its companions elements.) Most \reasonable" source models have an entropy rate h; namely, if x n 1 is randomly drawn according to the source model P, then the following limit exists, 1): For instance, the entropy rate of a sequence drawn according to the memoryless model equals the entropy of the distribution of individual characters. For a Markov chain with transition probabilities p i;j , the entropy rate is h =
منابع مشابه
Average Redundancy for Known Sources: Ubiquitous Trees in Source Coding∗
Analytic information theory aims at studying problems of information theory using analytic techniques of computer science and combinatorics. Following Hadamard’s precept, these problems are tackled by complex analysis methods such as generating functions, Mellin transform, Fourier series, saddle point method, analytic poissonization and depoissonization, and singularity analysis. This approach ...
متن کاملRedundancy of Lossless Data Compression for Known Sources by Analytic Methods
Lossless data compression is a facet of source coding and a well studied problem of information theory. Its goal is to find a shortest possible binary code that can be unambiguously recovered. In this paper focus on rigorous analysis of code redundancy for known sources. The redundancy rate problem determines by how much the actual code length exceeds the optimal code length. We present precise...
متن کاملAnalytic Variations on Redundancy Rates of Renewal Processes Analytic Variations on Redundancy Rates of Renewal Processes Analytic Variations on Redundancy Rates of Renewal Processes
Csisz ar and Shields have recently proved that the minimax redundancy for a class of renewal processes is ( p n) where n is the block length. This interesting result provides a rst non-trivial bound on redundancy for a non-parametric family of processes. The present paper provides a precise estimate up to the constant term of the redundancy rate for such sources. The asymptotic expansion is der...
متن کاملSet a bi-objective redundancy allocation model to optimize the reliability and cost of the Series-parallel systems using NSGA II problem
With the huge global and wide range of attention placed upon quality, promoting and optimize the reliability of the products during the design process has turned out to be a high priority. In this study, the researcher have adopted one of the existing models in the reliability science and propose a bi-objective model for redundancy allocation in the series-parallel systems in accordance with th...
متن کاملUsing NSGA II Algorithm for a Three Objectives Redundancy Allocation Problem with k-out-of-n Sub-Systems
in the new production systems, finding a way to improving the product and system reliability in design is a very important. The reliability of the products and systems may improve using different methods. One of this methods is redundancy allocation problem. In this problem by adding redundant component to sub-systems under some constraints, the reliability improved. In this paper we worked on ...
متن کامل